Retroactive information searching enabled by neural sensing

ABSTRACT

A method for retrieving information includes detecting a first neural activity of a user, wherein the first neural activity corresponds to a key thought of the user; in response to detecting the first neural activity, generating a search query based on speech occurring prior to detecting the first neural activity; retrieving information based on the search query; and transmitting the information to one or more output devices.

BACKGROUND Field of the Embodiments

Embodiments of the present disclosure relate generally to computerinformation systems, and, more specifically, to retroactive informationsearching enabled by neural sensing.

Description of the Related Art

Smart phones, wearables, and other wireless devices enable a user toaccess the Internet and other information sources to retrieveinformation on essentially any subject. Freed from the necessity of awired connection, a user can now perform an Internet search by opening aweb browser on a smartphone or electronic tablet whenever wirelessservice is available. In addition, the widespread availability ofintelligent personal assistants (IPAs), such as Microsoft Cortana™,Apple Siri™, and Amazon Alexa™, enable a user to initiate a search forinformation on a particular topic without even looking at a displayscreen or manually entering search parameters. Instead, the user canretrieve information from the Internet verbally by speaking a questionto the IPA.

However, for a user to pull out a smartphone or speak to a digitalassistant in the middle of a face-to-face conversation is generallyregarded as socially rude behavior. In addition, searching forinformation relevant to an on-going conversation diverts the user'sattention away from the conversation. Thus, by focusing on a particularaspect of the conversation and attempting to expand knowledge relevantto that aspect with an Internet search, the user can miss the overallthread of the conversation. Further, in the case of IPAs, verballydirecting a question to an IPA-enabled device interrupts theconversation itself. Thus, while modern wireless devices and theInternet provide immediate access to a great deal of specificinformation on virtually any subject, taking advantage of such access inreal-time and without disrupting an ongoing conversation can beproblematic.

In light of the above, more effective techniques for enabling a user todiscreetly access relevant information would be useful.

SUMMARY

The various embodiments set forth a method for providing informationbased on neural activity. The method includes detecting first neuralactivity of a user, wherein the first neural activity corresponds to akey thought of the user; in response to detecting the first neuralactivity, generating a search query based on speech occurring prior todetecting the first neural activity; receiving information based on thesearch query; and transmitting the information to an output device thatprivately outputs information to the user.

At least one advantage of the disclosed embodiments is that informationrelevant to an on-going conversation can be provided to a user inreal-time. A further advantage is that relevant information can beprovided to the user privately, for example, via headphones, earbuds,hearables, and/or a head-mounted display. Additionally, a search forrelevant information can be initiated and performed in the background.Accordingly, relevant information can be retrieved and provided to theuser without interrupting an on-going conversation.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

So that the manner in which the above recited features of the variousembodiments can be understood in detail, a more particular descriptionof the various embodiments, briefly summarized above, may be had byreference to embodiments, some of which are illustrated in the appendeddrawings. It is to be noted, however, that the appended drawingsillustrate only typical embodiments and are therefore not to beconsidered limiting of its scope, for the various embodiments may admitto other equally effective embodiments.

FIG. 1 is a schematic diagram illustrating a neural sensing-enabledinformation retrieval system, configured to implement one or moreaspects of the various embodiments of the present disclosure.

FIG. 2 is a more detailed illustration of a computing device in theneural sensing-enabled information retrieval system, according tovarious embodiments.

FIG. 3 is a schematic diagram illustrating a headphone system configuredto implement one or more aspects of the disclosure.

FIG. 4 is a schematic diagram illustrating a stereo earbud systemconfigured to implement various aspects of the present disclosure.

FIG. 5 sets forth a flowchart of method steps for providing informationto a user based on neural activity, according to various embodiments ofthe present disclosure.

FIG. 6 sets forth a flowchart of method steps for detecting a neuralactivity in a user when the user thinks a specific thought, according tovarious embodiments of the present disclosure.

FIG. 7 sets forth a flowchart of method steps for generating one or moresearch queries in response to a request from the user for moreinformation, according to various embodiments of the present disclosure.

FIG. 8 sets forth a flowchart of method steps for presenting selectedsearch result information to a user, according to various embodiments ofthe present disclosure.

For clarity, identical reference numbers have been used, whereapplicable, to designate identical elements that are common betweenfigures. It is contemplated that features of one embodiment may beincorporated in other embodiments without further recitation.

DETAILED DESCRIPTION

FIG. 1 is a schematic diagram illustrating a neural sensing-enabledinformation retrieval system 100, configured to implement one or moreaspects of the various embodiments of the present disclosure. Neuralsensing-enabled information retrieval system 100 is configured toprivately initiate a search for information related to an on-goingconversation in which a user is engaged, and then present information tothe user that is received in response to the search. Detection ofspecific neural activity in the user initiates the search, which isperformed retroactively on topics and/or questions included in aconversation that was recorded prior to the detection of the specificneural activity. The neural activity is associated with that particularuser when thinking a certain “key thought” in mind. Therefore, thesearch can be initiated unobtrusively and without interrupting theconversation. In addition, information received in response to thesearch can be presented via private audio, which also does not interruptthe conversation. Thus, neural sensing-enabled information retrievalsystem 100 effectively provides a user with “ambient omniscience” and“ambient knowledge injection” using a socially unobtrusive and subtleuser interface method—human thought.

Neural sensing-enabled information retrieval system 100 includes ahead-worn auditory device 120, a neural sensing apparatus 130, acomputing device 140, and, in some embodiments, a connectivity apparatus150. In some embodiments, head-worn auditory device 120, neural sensingapparatus 130, and/or computing device 140 are integrated into a singledevice, such as a headphone-based assembly. Alternatively, in someembodiments, head-worn auditory device 120, neural sensing apparatus130, and/or computing device 140 are each separate physical devices. Forexample, in one such embodiment, head-worn auditory device 120 andneural sensing apparatus 130 are integrated into an earbud or pair ofearbuds, while computing device 140 is implemented as a separatecomputing device, such as a smartphone or a wearable computing device.

Head-worn auditory device 120 can be any technically feasible head-wornor head-mountable device that includes an audio recording subsystem 121(e.g., a microphone) and an audio playback subsystem 122 (e.g., a loudspeaker). For example, in some embodiments, head-worn auditory device120 can be configured as a headphone-based assembly, such as supra-auralheadphones, which rest directly on the user's outer ear, or circumauralheadphones, which completely surround the ears. In other embodiments,head-worn auditory device 120 can be configured as anearpiece/microphone assembly or as a single or a pair of earbuds. Insome embodiments, head-worn auditory device 120 includes additionalsensors 123, such as one or more inertial measurement units (IMUS) orother electronic devices that measure and report motion. In suchembodiments, such additional sensors 123 can be coupled to the jaw of auser to enable detection of speech on the part of the user.

Audio recording subsystem 121 can include one or more microphonespositioned to optimize the recording of the user's voice and generateaudio data 101. Thus, audio recording subsystem 121 may includetraditional over-air microphone technologies, bone-conductionmicrophones, and/or other transducer-based technology that convertssound into an electrical signal. Alternatively or additionally, audiorecording subsystem 121 may include multiple microphones to enable beamforming, to further optimize the recording of a user's voice and/or thevoices of other persons engaged in a conversation with the user.

Audio playback subsystem 122 includes any technically feasibleloudspeakers or other audio playback device or devices that are worn bythe user. For example, in some embodiments, audio playback subsystem 122includes one or more in-ear, on-ear, over-ear, and/or bone-conductiveloudspeakers. Alternatively, in some embodiments audio playbacksubsystem 122 is configured to provide private audio at a distance, forexample, through speaker arrays (“hypersonic sound” and the like). Insuch embodiments, the loudspeakers of audio playback subsystem 122 maybe located elsewhere on the user's body, rather than in head-wornauditory device 120.

Neural sensing apparatus 130 can be any technically feasible system ordevice configured to measure the neural activity of a user via one ormore neural sensors 131. For example, in some embodiments, neuralsensors 131 include one or more electroencephalography (EEG) sensors orother neural sensors for monitoring the surface electrical activity of auser's brain in real time. Typically, neural sensing apparatus 130provides information about the user's neural activity via one or moreEEG sensors (such as wet or dry electrodes) that are in contact with theuser's scalp, face, or head at specific locations that are selected tomonitor one or more specific regions of interest in the brain. In someembodiments, neural sensors 131 of neural sensing apparatus 130 can beintegrated into head-worn auditory device 120 and/or can be worn as aseparate assembly. In some embodiments, neural sensing apparatus 130 isconfigured to perform preprocessing of signals received from neuralsensors 131, including noise reduction techniques, and to provide neuraldata 102 derived from such signals to computing device 140 for themonitoring of neural activity of the user and the detection of specificneural activity in the user. In some embodiments, neural sensingapparatus 130 is configured to record data derived from the signalsreceived from neural sensors 131, while in other embodiments computingdevice 140 is configured to record such data.

Connectivity apparatus 150 is configured to provide wirelessconnectivity to external devices and/or networks, thereby enablingneural sensing-enabled information retrieval system 100 to search onlinefor relevant questions and/or topics that have been extracted fromrecorded speech. As such, in some embodiments, connectivity apparatus150 includes a wireless transceiver 151, such as a Bluetooth®transceiver, a WiFi transceiver, a wireless Internet transceiver and/ora transceiver compatible with a broadband cellular data network orsatellite data network. Thus, in such embodiments, connectivityapparatus 150 enables connectivity to smartphones, electronic tablets,laptop computers, and the like, for additional computational resourcesand information. Alternatively or additionally, connectivity apparatus150 enables connectivity to one or more wireless networks for accessinginformation via the Internet. For example, connectivity apparatus 150enables connection to a remote server, where databases of informationthat is the subject of a search may be leveraged, and where training ofan application or algorithm can be off-loaded for subsequentidentification of the neural activity associated with the user thinkinga specific key thought. In some embodiments, neural sensing-enabledinformation retrieval system 100 connects via connectivity apparatus 150and a Bluetooth® connection to a user's smartphone, and routes searchqueries therethrough. Furthermore, in embodiments in which externaloutput devices are included in or available to neural sensing-enabledinformation retrieval system 100, connectivity apparatus 150 can sendinformation received in response to a search request to the appropriateexternal output device.

Computing device 140 is configured to perform various functions ofneural sensing-enabled information retrieval system 100, includinganalyzing neural data 102 to recognize the neural activity associatedwith the user thinking a key thought, receiving and converting audiodata 101 to text, performing textual content analysis, performinginformation searches, and presenting returned search information to theuser. In addition, computing device 140 may be configured to interactwith connectivity apparatus 150 to establish connection to the Internetand enable information searches and, in some embodiments, off-loadcomputationally expensive tasks to a cloud-computing system.

Computing device 140 can be incorporated into head-worn auditory device120. Alternatively, the functionality of computing device 140 can beincorporated into a mobile computing device, such as a suitablyprogrammed smartphone, electronic tablet, smart watch, or otherwearable. One embodiment of computing device 140 is described below inconjunction with FIG. 2.

FIG. 2 is a more detailed illustration of computing device 140,according to various embodiments. Computing device 140 is configured toimplement one or more aspects of the present disclosure describedherein. Computing device 140 may be any type of device capable ofexecuting application programs including, without limitation,instructions associated with a neural analysis application 201, an audioprocessing application 202, a speech-to-text conversion application 203,a language parsing program 204, and/or a search result presentationapplication 205. For example, and without limitation, computing device140 may be an electronic tablet, a smartphone, a laptop computer, etc.Alternatively, computing device 140 may be implemented as a stand-alonechip, such as a microprocessor, or as part of a more comprehensivesolution that is implemented as an application-specific integratedcircuit (ASIC), a system-on-a-chip (SoC), and so forth. Generally,computing device 140 may be configured to coordinate the overalloperation of a computer-based system, such as neural sensing-enabledinformation retrieval system 100. In other embodiments, computing device140 may be coupled to, but separate from such a computer-based system.In such embodiments, the computer-based system may include a separateprocessor that transmits data to computing device 140, such as audiodata 101 and/or neural data 102, and may be included in a consumerelectronic device, such as a personal computer, smartphone, orheadphone-based device. As shown, computing device 140 includes, withoutlimitation, a processor 210, input/output (I/O) devices 220, and amemory 230.

Processor 210 may be implemented as a central processing unit (CPU), agraphics processing unit (GPU), an application-specific integratedcircuit (ASIC), a field programmable gate array (FPGA), any other typeof processing unit, or a combination of different processing units. Ingeneral, processor 210 may be any technically feasible hardware unitcapable of processing data and/or executing software applications tofacilitate operation of neural sensing-enabled information retrievalsystem 100 of FIG. 1, as described herein. Among other things, andwithout limitation, processor 210 may be configured to executeinstructions associated with neural analysis application 201, audioprocessing application 202, speech-to-text conversion application 203,language parsing program 204, and/or search result presentationapplication 205.

Memory 230 may include a random access memory (RAM) module, a flashmemory unit, or any other type of memory unit or combination thereof,and may include a single memory module or a collection of memorymodules. As shown, in some embodiments, some or all of neural analysisapplication 201, audio processing application 202, speech-to-textconversion application 203, language parsing program 204, and/or searchresult presentation application 205 may reside in memory 230 duringoperation.

I/O devices 220 includes one or more devices capable of both receivinginput, such as a keyboard, a mouse, a touch-sensitive screen, amicrophone (including a microphone associated with audio recordingsubsystem 121) and so forth, as well as devices capable of providingoutput, such as a display screen, loudspeakers (including a loudspeakerassociated with audio playback subsystem 122), and the like. The displayscreen may be incorporated in neural sensing-enabled informationretrieval system 100 or may be external to neural sensing-enabledinformation retrieval system 100, such as a computer monitor, a videodisplay screen, a display apparatus incorporated into a separate handheld device, or any other technically feasible display screen orprojection device.

In operation, neural analysis application 201 analyzes neural data 102to recognize the neural activity associated with the user thinking a keythought. For example, in some embodiments, neural analysis application201 includes a neural activity detection algorithm. In some embodiments,neural analysis application 201 and/or computing device 140 receives aninitial training (described below in conjunction with FIG. 5), so thatthe neural activity associated with a particular user thinking aspecific key thought can be differentiated from other neural activityand, therefore, can be detected in real-time. Audio processingapplication 202 is employed by computing device 140 to performpost-processing on audio data 101 received from audio recordingsubsystem 121 to clean the audio signals included in audio data 101 andrender voice portions of such audio signals to be more audible. Forexample, in some embodiments, audio processing application 202 includesvoice isolation techniques and algorithms known in the art. Computingdevice 140 employs speech-to-text conversion application 203 forconverting audio data 101 to text for subsequent parsing and analysis.Language parsing program 204 is employed by computing device 140 foranalyzing textual content generated by speech-to-text conversionapplication 203 to determine questions, key words, and/or topics thatcan be included in search queries. Specifically, language parsingprogram 204 parses the textual content to detect explicit questionsasked by the user and/or to determine search terms via context,frequency of use, and other language cues. In some embodiments, languageparsing program 204 performs such parsing using common natural languageunderstanding (NLU) and natural language processing (NLP) methods, aswell as terminology extraction, discourse analysis, and the like. Searchresult presentation application 205 determines what is currently thepreferred presentation mode of the information that is received inresponse to the search. For example, presentation application 205determines the preferred presentation mode based on a specific input orselection from the user; whether information received in response to asearch can be presented audibly and/or visually; what output mechanismsare available; what the current cognitive load on the user is; and/orwhether the user is currently speaking.

In some embodiments, neural sensing-enabled information retrieval system100 is implemented in a headphone-based assembly. One such embodiment isillustrated in FIG. 3. FIG. 3 is a schematic diagram illustrating aheadphone system 300 configured to implement one or more aspects of thedisclosure. Headphone system 300 may include, without limitation, twoearcups 301, coupled to a headband 303 via a respective arm 302. Eachearcup 301 is configured to fit over the outer ear of a user, whenheadphone system 300 is worn by the user, and includes, among otherthings, an ear-surround cushion 304 coupled to a housing 305. In someembodiments, headphone system 300 may be configured with a singleearcup. Furthermore, in some embodiments, headphone system 300 may beconfigured as a supra-aural headphone system, while in otherembodiments, headphone system 300 may be configured as a circumauralheadphone system. In the embodiment illustrated in FIG. 3, headphonesystem 300 is configured as a circumaural headphone system.

As shown, head-worn auditory device 120 is incorporated into headphonesystem 300. Thus, included in earcups 301 are one or more microphones321 of audio recording subsystem 121 (not shown), one or moreloudspeakers 322 of audio playback subsystem 122 (not shown), and one ormore neural sensors 131 of neural sensing apparatus 130 (not shown). Inthe embodiment illustrated in FIG. 3, neural sensors 131 are located inear-surround cushions 304 to be in contact with specific locations onthe head of a user when headphone system 300 is worn by the user. Insome embodiments, neural sensors 131 are located in the headband 303. Insome embodiments, computing device 140 and/or connectivity module 150are disposed within housing 305 or headband 303.

When headphone system 300 is worn by a user, ear-surround cushion 304seals against the user's head, so that each earcup 301 forms an acousticcavity around one of the user's ears. By design, ear-surround cushion304 forms and acoustically isolates this acoustic cavity from thesurroundings for enhanced listening. In addition, neural sensors 131 arepositioned to be in contact with specific locations on the head of theuser, so that neural activity of the user can be measured and recorded.

In some embodiments, neural sensing-enabled information retrieval system100 is implemented in an earbud-based assembly. One such embodiment isillustrated in FIG. 4. FIG. 4 is a schematic diagram illustrating astereo earbud system 400 configured to implement various aspects of thepresent disclosure. Stereo earbud system 400 includes, withoutlimitation, a left earbud 401 and a right earbud 402, each coupled to aplug assembly 420 via a wired connection 403. Alternatively, left earbud401 and right earbud 402 may be configured as wireless earbuds. Stereoearbud system 400 may further include, without limitation, a volumecontrol module 404 coupled to left earbud 401, right earbud 402, andplug assembly 420 as shown. Stereo earbud system 400 further includes,without limitation, neural sensors 131 disposed on a surface of leftearbud 401 and/or right earbud 402. Thus, when a user has inserted leftearbud 401 and right earbud 402, neural sensors 131 are positioned incontact with respective locations within the ear canal of the user,thereby enabling the measurement and recording of neural activity of theuser. In addition, stereo earbud system 400 can further include, withoutlimitation, one or more microphones 421.

In some embodiments, stereo earbud system 400 further includes, withoutlimitation, one or more components of computing device 140 and/orconnectivity apparatus 150. In such embodiments, the components ofcomputing device 140 and/or connectivity apparatus 150 can beincorporated into left earbud 401, right earbud 402, and/or volumecontrol module 404. Alternatively, components of computing device 140and/or connectivity apparatus 150 can be incorporated into a deviceexternal to stereo earbud system 400. Alternatively, in someembodiments, stereo earbud system 400 can be configured as a mono earbudsystem with a single earbud or with two earbuds that produce identicalaudio output.

According to various embodiments, neural sensing-enabled informationretrieval system 100 is configured to detect a user request forinformation while the user is engaged in a conversation, parse theverbal content of speech in the conversation that precedes the userrequest, search for relevant information and/or answers to questions,and deliver the information to the user. Such embodiments are describedbelow in conjunction with FIGS. 5-8.

FIG. 5 sets forth a flowchart of method steps for searching forinformation, according to various embodiments of the present disclosure.Although the method steps are described with respect to the systems ofFIGS. 1-4, persons skilled in the art will understand that any systemconfigured to perform the method steps, in any order, falls within thescope of the various embodiments.

Prior to the method steps, computing device 140 and/or neural analysisapplication 201 undergoes a training process for detection of specificneural activity when a particular user is thinking a key thought. Forexample, in some embodiments the training process is performed as partof initial setup of neural sensing-enabled information retrieval system100. In some embodiments, the training process includes measurement vianeural sensors 131 of a user baseline state, in which the user attemptsto think about nothing. Neural sensing-enabled information retrievalsystem 100 then prompts the user to think about a specific key thoughtfor a set amount of time. This may include thinking of a word, color,object, piece of music, motion or action, etc. Generally, to reduce thelikelihood of false triggers, the key thought selected by the user issomething specific and unusual. After a sufficient time interval ofthinking the key thought, neural analysis application 201 then analyzesand compares the recorded neural data 102 for each state.Differentiating patterns and features between the recorded neural data102 for the baseline state and the key thought state are then used tocreate a classification or neural pattern that is associated with theuser thinking the key thought. The classification or neural pattern canthen be used to identify when the user is thinking of that particularkey thought during operation of neural sensing-enabled informationretrieval system 100.

In some embodiments, computing device 140 and/or neural analysisapplication 201 can be trained to recognize multiple different neuralactivities in the user, where each neural activity is associated withthe user thinking a different respective key thought. In suchembodiments, multiple different commands can be directed to neuralsensing-enabled information retrieval system 100 by the user, ratherthan simply requesting a search for information pertinent to an on-goingconversation. In some embodiments, the computational resources employedin such a training process can be off-loaded to a cloud-computingsystem.

As shown, a method 500 begins at optional step 501, in which audiorecording subsystem 121 detects that the user is speaking. In responseto detecting that the user is speaking, computing device 140 bringsneural sensing-enabled information retrieval system 100 from a low-powerstate to a higher-power state. Specifically, in the low-power state,neural sensing apparatus 130 is powered off and one or more microphonesincluded in audio recording subsystem 121 are on and generating audiodata 101. In some embodiments, audio recording subsystem 121 records asmall quantity of audio data 101 in a buffer sufficient to determinewhether audio data 101 includes speech or other verbal utterances fromthe user, for example two to ten seconds of audio data 101.Alternatively, in some embodiments, recording subsystem 121 does notrecord any audio data 101 at all in the low-power state.

In some embodiments, audio recording system 121 detects that the user isspeaking based on a sound volume of detected sound exceeding a minimumthreshold level, since the voice of the user should be louder than thatof others engaged in conversation with the user. Alternatively oradditionally, in some embodiments, audio recording system 121 detectsthat the user is speaking based on another input, such as the detectionof motion of the jaw of the user. In such embodiments, audio recordingsystem 121 receives such an input from an IMU or other additionalsensor(s) 123 included in head-worn auditory device 120.

In step 502, audio recording subsystem 121 operates in the higher-powerstate, and audio recording subsystem 121 records audio data 101 forsubsequent analysis. In embodiments in which optional step 501 isincluded in method 500, audio recording subsystem 121 switches to thehigher-power state in step 502. In embodiments in which optional step501 is not included in method 500, audio recording subsystem 121constantly operates in the higher-power state in step 502.

Generally, in the higher-power state, audio recording subsystem 121records audio data 101 for a sufficient duration of time to enablesubsequent detection of questions and/or key words that can be employedas search queries. For example, in some embodiments, audio recordingsubsystem 121 records up to about one minute to about five minutes ofaudio data 101 before overwriting older audio data 101. Generally, audiodata 101 includes speech and/or other verbal utterances of the user and,in some embodiments, speech and/or verbal utterances of others engagedin conversation with the user. In some embodiments, audio processingapplication 202 is employed, for example by computing device 140, toperform post processing on audio signals included in audio data 101 toclean such signals and to extract the voice component or components.Furthermore, in some embodiments, audio recording subsystem 121 includesmetadata for portions of audio data 101 indicating when the user isspeaking, so that subsequent analysis of audio data 101 can be performedseparately on user speech and on the speech of other participants in theconversation. Techniques for detecting whether the user is speaking ornot is described above in conjunction with step 501.

In step 503, which is performed in parallel with step 502, neuralsensing apparatus 130 operates in the higher-power state, and is poweredon so that neural activity of the user can be monitored via neuralsensors 131. In embodiments in which optional step 501 is included inmethod 500, neural sensing apparatus 130 is powered on in step 503. Inembodiments in which optional step 501 is not included in method 500,neural sensing apparatus 130 constantly operates in the higher-powerstate in step 503.

In step 503, neural sensing apparatus 130 monitors user neural activityand detects neural activity in the user associated with the userthinking a specific key thought. Various embodiments of step 503 aredescribed in conjunction with FIG. 6.

FIG. 6 sets forth a flowchart of method steps for detecting a neuralactivity in a user when the user thinks a specific thought, according tovarious embodiments of the present disclosure.

In step 601, computing device 140 causes neural sensing apparatus 130 tobe powered on, for example, in response to detecting that the user hasbegun speaking. Neural sensing apparatus 130 then begins generatingneural data 102. In some embodiments, neural data 102 are generatedbased on electrical measurements from neuron firings in the brain of theuser. More specifically, in such embodiments, the electricalmeasurements are measurements of the neural activity occurring in one ormore regions of the brain, rather than the electrical activity of aspecific neuron or neurons in the brain of the user. Thus, the neuralactivity in the brain measured in step 601 is more than the neuralactivity associated with the firing of specifically targeted neurons.Instead, the neural activity measured is associated with an ensemble oraggregate activity in one or more specialized regions of brain. Inaddition, in such embodiments, the neural activity associated withmultiple regions of the brain, and interactions therebetween, aremeasured. Thus, various components of neural activity associated withdifferent functional regions of the brain can be monitored in step 601,such as certain emotional components, memory components, a componentassociated with mental effort currently being exerted by the user, etc.Therefore, the neural activity measured in step 601 varies depending onwhat specific thought or thoughts are currently in the mind of the user,and is not merely the measurement of the amplitude of one or more typesof brainwaves, such as theta waves (4-7 Hz), alpha waves (8-15 Hz), betawaves (16-31 Hz), gamma waves (32-100 Hz), delta waves (0.1-3 Hz), andthe like. For example, in some embodiments, neural data 102 includestime-transient information, and is therefore not limited to a repeatingpattern of brainwaves.

In step 602, computing device 140 receives neural data 102 from neuralsensing apparatus 130. Neural data 102 can be transmitted wirelessly tocomputing device 140 or via a wired connection. In some embodiments,neural data 102 received in step 602 are associated with a specific timeperiod, for example the most recent few seconds of the currentconversation. Alternatively, neural data 102 received in step 602 may becontinuously streamed to computing device 140. In step 603, computingdevice 140 temporarily stores neural data 102, for example in a rotatingbuffer.

In step 604, computing device 140 analyzes the neural data 102 stored instep 604. In some embodiments, computing device 140 determines whetherneural data 102 includes specific neural activity or pattern that ismapped to or otherwise associated with the user thinking a key thought.In particular, computing device 140 determines whether neural data 102include the specific neural activity for which computing device 140and/or neural analysis application 201 have undergone training to detectprior to method 500 being performed. In some embodiments, the specificneural activity is not limited to a repeating pattern and includes oneor more time-transient components, since the act of thinking a keythought can include different components that vary over time. Forexample, the specific neural activity can include, without limitation, amemory component, an emotional component, a mental effort component, andthe like, each of which can be engaged at a different intensity level atdifferent times in the process of thinking a key thought. In someembodiments, the execution of neural analysis application 201 can beoff-loaded to a cloud-computing system. If the stored neural data 102include the specific neural activity, method 600 proceeds to step 504 inmethod 500; if the stored neural data 102 do not include the specificneural activity, method 600 proceeds back to step 602.

Returning to FIG. 5, in step 504 of method 500, computing device 140converts audio data 101 to text. In some embodiments, speech-to-textconversion application 203 is employed, for example by computing device140, to convert audio data 101 to text for subsequent parsing. In someembodiments, the execution of speech-to-text conversion application 203can be off-loaded to a cloud-computing system.

In step 505, computing device 140 generates one or more search queriesfor information that is beneficial to the user in the context of thecurrent conversation. Specifically, in some embodiments, computingdevice 140 performs analysis of the textual content generated in step504 to determine what specific information the user has requested and/orfor what topics the user can beneficially receive additionalinformation. Various embodiments of step 505 are described inconjunction with FIG. 7.

FIG. 7 sets forth a flowchart of method steps for generating one or moresearch queries in response to a request from the user for moreinformation, according to various embodiments of the present disclosure.

In step 701, computing device 140 determines whether the user has posedor verbalized one or more questions in the recent past. Specifically,computing device 140 determines whether the user's speech, during anappropriate time interval includes one or more questions. Generally, theappropriate time interval includes the time immediately prior todetermining that neural data 102 includes a specific neural activity orpattern in step 604. For example, in some embodiments, the appropriatetime interval equals the duration of time during which audio recordingsubsystem 121 has recorded audio data 101. Alternatively, in someembodiments, the appropriate time interval equals the most recentlyrecorded portion of audio data 101, but not all of the recorded audiodata 101. For example, in such an embodiment, audio data 101 includesthe most recent five minutes of the current conversation, while theappropriate time interval equals the most recently recorded one minuteof the current conversation. In such embodiments, if no question isdetected in the appropriate time interval, a greater portion of theaudio data can then be searched for a question. That is, the appropriatetime interval is expanded or shifted earlier in time. If computingdevice 140 determines that the user's speech includes one or morequestions in step 701, method 700 proceeds to step 704; if not, method700 proceeds to step 702. In some embodiments, language parsing program204 is employed by computing device 140 to perform the analysis oftextual content in step 701. Alternatively, in some embodiments, theexecution of language parsing program 204 can be off-loaded to acloud-computing system.

In step 702, computing device 140 determines whether the speech of otherspeakers or participants in the conversation during the appropriate timeinterval includes one or more questions. As noted above, in someembodiments, a shorter time interval may first be employed in step 702to search for questions, followed by a longer time interval. Ifcomputing device 140 determines that the speech of other speakers orparticipants in the conversation during the appropriate time intervalincludes one or more questions, method 700 proceeds to step 704; if no,method 700 proceeds to step 703. In some embodiments, language parsingprogram 204 is employed by computing device 140 to perform the analysisof textual content in step 702.

In step 703, which is performed in response to no explicit questionsbeing detected in the most recent portion of the current conversation,computing device 140 performs a more thorough search of the recordedtext from the current conversation to generate one or more searchqueries. In some embodiments, the search can be over a first timeinterval that occurs immediately before the key thought is detected (instep 503 of method 500), then over one or more additional time intervalsthat have occurred increasingly earlier in the current conversation.Computing device 140 identifies a key or primary topic and/or multipletopics that are included in the text corresponding to the first timeinterval and/or the additional time intervals. In some embodiments, textof the user's speech is employed in the search, while in otherembodiments, text of the speech of other participants is also employedin the search. As noted above, language parsing program 204 generallyparses the content of text generated in step 504 of method 500 to detectsearch terms via context, frequency of use, and other language cues. Forexample, if the user is having a conversation about politics and beginsdiscussing President Trump's stance on renewable energy, the system willidentify (through keywords, context, and other cues) the key themes ofthe portion of the current conversation preceding the request forinformation initiated by the user by thinking the key thought.Therefore, the keywords extracted for the search may be: “president,”“Trump,” “renewable energy,” and “stance.” These extracted search termsare employed as search queries in step 704.

In step 704, computing device 140 generates one or more search queriesbased on the questions detected in step 701 or 702, or on the searchterms extracted in step 703. Method 700 then proceeds to step 506 ofmethod 500. In the embodiment illustrated in FIG. 7, search queries aregenerated in step 704 based on one of questions verbalized by the user,questions verbalized by other speakers, or search terms extracted from arecorded portion of the conversation. Alternatively, in someembodiments, search queries generated in step 704 are based on anycombination of questions verbalized by the user, questions verbalized byother speakers, and/or search terms extracted from a recorded portion ofthe conversation.

Returning to FIG. 5, in step 506 of method 500, computing device 140performs one or more information searches using the search queriesgenerated in step 704. As noted, the search queries can be based onexplicit questions recently posed in the current conversation and/or ona topic or topics discussed in the current conversation. In someembodiments, the search or searches are performed locally, for examplein a local database of information. Alternatively or additionally, insome embodiments, the search or searches are performed via a network,including the Internet.

In step 507, computing device 140 aggregates the search resultsreceived. For example, in some embodiments, computing device 140 ranksor arranges search results in order of relevance, eliminates redundantsearch results, and selects which search results received in response toperforming the search of step 506 are to be presented to the user.

In some embodiments, the selection of content by computing device 140may be determined by the output modality that will ultimately presentthe information to the user. Thus, in such embodiments, as part of theinformation aggregation process of step 507, computing device 140determines what information output modalities are currently available tothe user and discards or transforms information that cannot be presentedto the user in the current state. For example, when only an audio outputmodality is available for presenting information to the user, such asloudspeakers included in audio playback subsystem 122, computing device140 discards image-based information received as a search result, andcan convert text-based information received into speech, via atext-to-voice application. In another example, an audio output modalitymay not be currently available for presenting information to the userbut a visual output modality may be available, such as when computingdevice 140 has detected that the user is speaking or is listening toanother participant, and neural sensing-enabled information retrievalsystem 100 includes a head-mounted display device. In such an example,computing device 140 can convert speech-containing audible informationto text via speech-to-text conversion application 203, and present thetext to the user via the head-mounted display device.

In step 508, computing device 140 presents selected search resultinformation to the user. In some embodiments, search result presentationapplication 205 is employed, for example by computing device 140, toconvert audio data 101 to text for subsequent parsing. In someembodiments, the execution of search result presentation application 205can be off-loaded to a cloud-computing system.

In some embodiments, how and when computing device 140 presents selectedsearch result information to the user may depend on one or more factors.A selection of such embodiments of step 508 are described in conjunctionwith FIG. 8. FIG. 8 sets forth a flowchart of method steps forpresenting selected search result information to a user, according tovarious embodiments of the present disclosure.

In step 801, computing device 140 determines or looks up the outputmodalities of neural sensing-enabled information retrieval system 100that are currently available, including audio output modalities andvisual output modalities. For example, in embodiments in which neuralsensing-enabled information retrieval system 100 is implemented in aheadphone-based assembly or an earbud-based assembly, one audio outputmodality of neural sensing-enabled information retrieval system 100 is aprivate audio mode, in which only the user can hear the presentation ofsearch result information. In embodiments in which loudspeakers externalto neural sensing-enabled information retrieval system 100 are availableto receive audio information from computing device 140, such as a smartspeaker or electronic tablet wirelessly connected to neuralsensing-enabled information retrieval system 100, another audio outputmodality is public audio mode, in which other participants in theconversation can also hear the presentation of search resultinformation. In embodiments in which neural sensing-enabled informationretrieval system 100 includes a head-mounted display device, one visualoutput modality of neural sensing-enabled information retrieval system100 is a private visual mode, in which only the user can see thepresentation of search result information. In embodiments in which adisplay device external to neural sensing-enabled information retrievalsystem 100 is available to receive image-based and/or video informationfrom computing device 140, such as a computer display screen or digitalprojection system wirelessly connected to neural sensing-enabledinformation retrieval system 100, another visual output modality ispublic visual mode, in which other participants in the conversation canalso see the presentation of search result information.

In step 802, computing device 140 determines or looks up what, if any,user output device preferences have been selected by the current user.In such embodiments, upon start-up and/or set-up of neuralsensing-enabled information retrieval system 100, the user may beprompted to select one or more default output preferences. Examples ofdefault output preferences include, without limitation, pausing audiopresentation of search result information when the user is speaking;pausing audio presentation of search result information when anyoneinvolved in the current conversation is speaking; providing searchresult information immediately; employing one of private audio mode(only audible to the user themselves, resulting in a “whispering” voiceinterface), public audio mode (audible to the user and people inproximity of the user), private video mode, or public video mode as thedefault output mode; and the like.

In step 803, computing device 140 determines the most appropriate outputdevice for the search result information to be presented. The mostappropriate output device selected by computing device 140 can be basedon multiple factors. Examples of such factors include, withoutlimitation, the output modalities determined to be available in step801; user selections determined in step 802; the content of the searchresult information (i.e., audio or visual); and the like.

In step 804, computing device 140 determines the most appropriate timeto present the search result information. The most appropriate timedetermined by computing device 140 can be based on multiple factors.Examples of such factors include, without limitation, user selectionsdetermined in step 802; whether the user is speaking; whether otherparticipants in the conversation are speaking; complexity of searchresult information to be presented (including quantity of and number ofoutput modes associated with the search result information); currentcognitive load on the user (based on information in neural data 102,such as indications that the user listening to another participant inthe conversation); whether the appropriate output device is currentlybeing used for a different task; and the like.

In step 805, computing device 140 determines whether the mostappropriate output device selected in step 803 is available at the mostappropriate time determined in step 804. If yes, method 800 proceeds tostep 806; if no, method 800 repeats step 805.

In step 806, computing device 140 presents the search result informationvia the most appropriate output device to the user or to allparticipants in the conversation, as appropriate. In addition, computingdevice 140 presents the search result information at the mostappropriate time.

In some embodiments of method 500, search queries are generated,searches performed, and search results aggregated via an intelligentpersonal assistants (IPAs), such as Microsoft Cortana™, Apple Siri™, andAmazon Alexa™. Thus, in such embodiments, steps 505, 506, and 507 (andin some embodiments also step 504) are performed by an IPA. For example,when computing device 140 determines that stored neural data 102 includethe specific neural activity, in lieu of performing step 504 (“Convertaudio data to text”), computing device 140 transmits an appropriaterecorded snippet, e.g., audio data 101, to a suitable IPA. The IPA thenperforms functions roughly equivalent to steps 505, 506, and 507, andreturns aggregated or otherwise filtered and organized search results tocomputing device 140. Computing device 140 then presents relevant searchresult information to the user, as described above in step 508.

Alternatively, in some embodiments, before transmitting audio data 101to the IPA, computing device 140 performs certain pre-analysis and/ortrimming, clipping, or other modifications on audio data 101, in lieu ofstep 504 (“Convert audio data to text”). For example, when a key thoughtis detected in step 503, computing device 140 then detects segments ofrecorded audio within audio data 101 which have the characteristics ofspoken language questions. In some embodiments, sentence boundaries arefirst detected, for example by finding pauses in the audio signal, thenquestions are detected by finding certain pitch contours that indicate aquestion in the language being spoken. For example, in English, risingvoice pitch at the end of a sentence generally indicates a question isbeing asked. Thus, in such embodiments, computing device 140 can employboth pause detection and pitch detection, which are different fromspeech recognition, to detect questions to be forwarded to the IPA.

Embodiments of method 500 enable a user to quickly and easily initiate asearch for information relevant to a currently on-going discussionwithout the interruption associated with consulting a smartphone orspeaking to a digital personal assistant. In addition, the embodimentsenable the search result information to be selectively received eitherprivately or publicly. For example, in one instance, a user is in anengineering brainstorm with a team from another company. Theconversation so far has been on topics that the user has prepared for,or already has knowledge of, but then the conversation shifts to an areathe user is less knowledgeable about. Thus, the user would benefit fromadditional information about the topic being discussed, but cannotaccess a computer or smartphone or open a dialogue with a voice agentduring the meeting without interrupting the current conversation.Instead, the user responds to a question to the best of his or herknowledge, then immediately thinks or holds in mind the key thought forwhich neural sensing-enabled information retrieval system 100 has beentrained to detect the associated neural activity. Neural sensing-enabledinformation retrieval system 100 detects the neural activity associatedwith the key thought, parses the content of recent user speech (and insome embodiments other speech), and performs a search related to thetopic being discussed. By the time the user has finished talking, neuralsensing-enabled information retrieval system 100 has found and is readyto deliver relevant search result information. In one embodiment, theinformation is immediately displayed privately to the user on ahead-mounted display device worn by the user. In another embodiment, therelevant search result information is “whispered” privately to the uservia one or two earbud speakers as soon as the user has stopped talking.Thus, the user can then immediately begin reviewing (or reading aloud)some or all of the relevant search result information.

In some embodiments, neural sensing-enabled information retrieval system100 can be configured to perform a different operation than initiating asearch in response to detecting a specific neural activity associatedwith a key thought. For example, in one such embodiment, neuralsensing-enabled information retrieval system 100 can be configured torecord an audio snippet of the current conversation for subsequenttranscription. In another such embodiment, neural sensing-enabledinformation retrieval system 100 can be configured to perform afact-checking function in response to detection a specific neuralactivity in the user. Thus, rather than determining whether recentspeech of participants in the conversation includes one or morequestions, computing device 140 parses a recent portion of theconversation for answered questions and/or stated facts, and then checksthe validity thereof via a search for relevant information. In someembodiments, the fact-checking function is performed on questionsanswered and/or facts stated by the user. Alternatively or additionally,the fact-checking function is performed on questions answered and/orfacts stated by other participants in the conversation.

In the embodiments described above in conjunction with FIGS. 5-8, neuralsensing-enabled information retrieval system 100 is configured tooperate in response to detecting a single neural activity that isassociated with a key thought being held in the mind of a particularuser. In other embodiments, neural sensing-enabled information retrievalsystem 100 can be configured to operate in response to detecting one ofmultiple different neural activities, each of which is associated with adifferent respective key thought being held in the mind of oneparticular user. In such embodiments, a training process for computingdevice 140 and/or neural analysis application 201 includes the detectionof a specific neural activity for each of the different key thoughtsbeing held in the mind of that particular user. In such embodiments,each of the key thoughts can be contrasted against a user baselinestate, for example when the user attempts to think about nothing.Furthermore, in such embodiments, computing device 140 can be configuredto perform a different action or process in response to detecting eachof the different neural activities. For example, in some embodiments,detection of a first key thought causes computing device 140 toprivately initiate a search for information relevant to the currentconversation, as described above; detection of a second key thoughtcauses computing device 140 to change a default output modality (forexample, presenting relevant search information only when the user isnot speaking); detection of a third key thought causes computing device140 select a particular output device as the default output device;detection of a fourth key thought causes computing device 140 to performa fact-checking function; detection of a fifth key thought causescomputing device 140 to record an audio snippet of the currentconversation for subsequent transcription; and so on.

In sum, various embodiments set forth systems and techniques forsearching for information in real time. In response to detecting aneural activity in a user associated with the user thinking a specifickey thought, a computing device generates one or more search queries andinitiates a search using the one or more search queries. The searchqueries are based on verbal utterances from a conversation that arerecorded prior to detecting the specific neural activity. Informationreceived as a result of the search queries can be presented privately tothe user or publicly to other participants in the conversation.

At least one technological improvement of the disclosed embodiments isthat a user can privately initiate a search for information relevant toan on-going conversation in real time and without interrupting anon-going conversation. A further technological improvement is thatrelevant search result information can be delivered seamlessly andprivately to the user, in real time and without interrupting an on-goingconversation. Yet another technological improvement is that the user canbe provided with the appearance of “ambient omniscience” to the outsideworld, since the user can privately and unobtrusively access informationthat is available online during an on-going conversation.

1. In some embodiments, a non-transitory computer-readable storagemedium includes instructions that, when executed by one or moreprocessors, configure the one or more processors to retrieve informationby performing the steps of: detecting first neural activity of a user,wherein the first neural activity corresponds to a key thought of theuser; in response to detecting the first neural activity, generating asearch query based on speech occurring prior to detecting the firstneural activity; receiving information based on the search query; andtransmitting the information to one or more output devices.

2. The non-transitory computer-readable storage medium of clause 1,wherein detecting the first neural activity comprises: receiving anoutput from a neural sensor that measures neural activity of the user;and identifying the first neural activity in the output.

3. The non-transitory computer-readable storage medium of clauses 1 or2, wherein the neural sensor includes an electroencephalography (EEG)device.

4. The non-transitory computer-readable storage medium of any of clauses1-3, further including instructions that, when executed by one or moreprocessors, configure the one or more processors to perform the step oftraining a neural analysis application to detect the first neuralactivity while the user is thinking the key thought, wherein the firstneural activity is detected by the neural analysis application.

5. The non-transitory computer-readable storage medium of any of clauses1-4, wherein detecting the first neural activity is performed by acloud-based computing system.

6. The non-transitory computer-readable storage medium of any of clauses1-5, further including instructions that, when executed by one or moreprocessors, configure the one or more processors to perform the step ofdetermining a preferred presentation mode, wherein the one or moreoutput devices correspond to the preferred presentation mode.

7. The non-transitory computer-readable storage medium of any of clauses1-6, wherein determining the preferred presentation mode comprises oneor more of determining a default presentation mode of the information,determining one or more hardware devices that are currently availablefor outputting the information, determining whether the information hasa visual component, and determining whether the information has an audiocomponent.

8. The non-transitory computer-readable storage medium of any of clauses1-7, further including instructions that, when executed by one or moreprocessors, configure the one or more processors to further perform thesteps of: determining that the user is speaking; and in response,initiating recording of speech, wherein determining that the user isspeaking comprises at least one of detecting motion of a jaw of the userand detecting speech at a volume exceeding a threshold level.

9. The non-transitory computer-readable storage medium of any of clauses1-8, wherein the speech includes at least one of speech by the user andspeech by a person in conversation with the user.

10. The non-transitory computer-readable storage medium of any ofclauses 1-9, wherein generating the search query based on the speechrecorded prior to detecting the first neural activity comprises:converting the speech to text; and extracting search terms from thetext, wherein the search query includes the one or more search terms.

11. The non-transitory computer-readable storage medium of any ofclauses 1-10, wherein transmitting information received in response tothe search to the one or more output devices comprises: determining atime indicated by the user to transmit the information received inresponse to the search; and transmitting the information received inresponse to the search at the time indicated by the user to one or moreoutput devices.

12. The non-transitory computer-readable storage medium of any ofclauses 1-11, wherein transmitting information received in response tothe search to the one or more output devices comprises: determining anoutput time that comprises one of a time when no speech is detected, atime immediately after receiving the information in response to thesearch, and a time when a cognitive load on the user is less than athreshold level; and transmitting the information received in responseto the search at the output time.

13. The non-transitory computer-readable storage medium of any ofclauses 1-12, wherein transmitting the information received in responseto the search to the one or more output devices comprises one oftransmitting the information received in response to the search to anoutput device that privately outputs information to the user andtransmitting the information received in response to the search to anoutput device that publicly outputs information.

14. In some embodiments, a method for providing information based onneural activity comprises: detecting first neural activity of a user,wherein the first neural activity corresponds to a key thought of theuser; in response to detecting the first neural activity, generating asearch query based on speech occurring prior to detecting the firstneural activity; receiving information based on the search query; andtransmitting the information to an output device that privately outputsinformation to the user.

15. The method of clause 14, wherein generating the search query basedon the speech recorded prior to detecting the first neural activitycomprises: converting the speech to text; and identifying one or morequestions in the text.

16. The method of clauses 14 or 15, further comprising: detecting asecond neural activity of the user; and in response to detecting thefirst neural activity, performing one of a fact-checking function onspeech occurring prior to detecting the second neural activity andrecording speech occurring after detecting the second neural activity.

17. In some embodiments, a system comprises: a neural sensor included ina head-worn assembly; and a processor coupled to the neural sensor andconfigured to: receive neural data from the neural sensor; detect, basedon the neural data, a first neural activity of a user, wherein the firstneural activity corresponds to a key thought of the user; in response todetecting the first neural activity, generate a search query based onspeech occurring prior to detecting the first neural activity; receiveinformation based on the search query; and transmit the information toan output device.

18. The system of clause 17, wherein the neural sensor includes anelectroencephalography (EEG) device.

19. The system of clauses 17 or 18, wherein the head-worn assemblycomprises one of a headphone-based assembly and an earbud-basedassembly.

20. The system of any of clauses 17-19, wherein the output devicecomprises one of a loudspeaker included in the head-worn assembly and adisplay device included in the head-worn assembly.

Any and all combinations of any of the claim elements recited in any ofthe claims and/or any elements described in this application, in anyfashion, fall within the contemplated scope of the present invention andprotection.

The descriptions of the various embodiments have been presented forpurposes of illustration, but are not intended to be exhaustive orlimited to the embodiments disclosed. Many modifications and variationswill be apparent to those of ordinary skill in the art without departingfrom the scope and spirit of the described embodiments.

Aspects of the present embodiments may be embodied as a system, methodor computer program product. Accordingly, aspects of the presentdisclosure may take the form of an entirely hardware embodiment, anentirely software embodiment (including firmware, resident software,micro-code, etc.) or an embodiment combining software and hardwareaspects that may all generally be referred to herein as a “module” or“system.” In addition, any hardware and/or software technique, process,function, component, engine, module, or system described in the presentdisclosure may be implemented as a circuit or set of circuits.Furthermore, aspects of the present disclosure may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

Aspects of the present disclosure are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, enable the implementation of the functions/acts specified inthe flowchart and/or block diagram block or blocks. Such processors maybe, without limitation, general purpose processors, special-purposeprocessors, application-specific processors, or field-programmableprocessors or gate arrays.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

While the preceding is directed to embodiments of the presentdisclosure, other and further embodiments of the disclosure may bedevised without departing from the basic scope thereof, and the scopethereof is determined by the claims that follow.

What is claimed is:
 1. A non-transitory computer-readable storage mediumincluding instructions that, when executed by one or more processors,configure the one or more processors to retrieve information byperforming the steps of: detecting speech; detecting a first neuralactivity of a user via one or more neural sensors that measure neuralactivity of one or more brain regions of the user; determining, via aneural analysis application, that the user is thinking a key thoughtbased on the detected first neural activity, wherein the neural analysisapplication analyzes the first neural activity to determine that thefirst neural activity is associated with the user thinking the keythought, wherein the neural analysis application is configured todifferentiate the first neural activity from other neural activity ofthe user; in response to determining that the user is thinking the keythought, generating a search query based on the speech occurring duringa predetermined time period prior to determining that the user isthinking the key thought; receiving information based on the searchquery; and transmitting the information to one or more output devices.2. The non-transitory computer-readable storage medium of claim 1,wherein detecting the first neural activity comprises: receiving anoutput from the one or more neural sensors that measure neural activityof the user; and identifying the first neural activity in the output. 3.The non-transitory computer-readable storage medium of claim 2, whereinthe one or more neural sensors includes one or moreelectroencephalography (EEG) devices.
 4. The non-transitorycomputer-readable storage medium of claim 2, wherein determining thatthe user is thinking the key thought is performed by a cloud-basedcomputing system executing the neural analysis application.
 5. Thenon-transitory computer-readable storage medium of claim 1, furtherincluding instructions that, when executed by the one or moreprocessors, configure the one or more processors to perform the step ofdetermining a preferred presentation mode, wherein the one or moreoutput devices correspond to the preferred presentation mode.
 6. Thenon-transitory computer-readable storage medium of claim 5, whereindetermining the preferred presentation mode comprises one or more ofdetermining a default presentation mode of the information, determiningone or more hardware devices that are currently available for outputtingthe information, determining whether the information has a visualcomponent, or determining whether the information has an audiocomponent.
 7. The non-transitory computer-readable storage medium ofclaim 1, further including instructions that, when executed by the oneor more processors, configure the one or more processors to furtherperform the steps of: determining that the user is speaking; and inresponse, initiating recording of the speech, wherein determining thatthe user is speaking comprises at least one of detecting motion of a jawof the user or detecting speech at a volume exceeding a threshold level.8. The non-transitory computer-readable storage medium of claim 7,wherein the speech includes at least one of speech by the user or speechby a person in conversation with the user.
 9. The non-transitorycomputer-readable storage medium of claim 1, wherein generating thesearch query based on the speech occurring prior to determining that theuser is thinking the key thought comprises: converting the speech totext; and extracting one or more search terms from the text, wherein thesearch query includes the one or more search terms.
 10. Thenon-transitory computer-readable storage medium of claim 1, whereintransmitting information received in response to the search query to theone or more output devices comprises: determining a time indicated bythe user to transmit the information received in response to the searchquery; and transmitting the information received in response to thesearch query at the time indicated by the user to the one or more outputdevices.
 11. The non-transitory computer-readable storage medium ofclaim 10, wherein transmitting information received in response to thesearch query to the one or more output devices comprises: determining anoutput time that comprises one of a time when no speech is detected, atime immediately after receiving the information in response to thesearch query, or a time when a cognitive load on the user is less than athreshold level; and transmitting the information received in responseto the search query at the output time.
 12. The non-transitorycomputer-readable storage medium of claim 1, wherein transmitting theinformation received in response to the search query to the one or moreoutput devices comprises one of transmitting the information received inresponse to the search query to an output device that privately outputsinformation to the user or transmitting the information received inresponse to the search query to an output device that publicly outputsinformation.
 13. A method for providing information based on neuralactivity, the method comprising: detecting speech; detecting a firstneural activity of a user via one or more neural sensors that measureneural activity of one or more brain regions of the user; determining,via a neural analysis application, that the user is thinking a keythought based on the detected first neural activity, wherein the neuralanalysis application analyzes the first neural activity to determinethat the first neural activity is associated with the user thinking thekey thought, wherein the neural analysis application is configured todifferentiate the first neural activity from other neural activity ofthe user; in response to determining that the user is thinking the keythought, generating a search query based on the speech occurring duringa predetermined time period prior to determining that the user isthinking the key thought; receiving information based on the searchquery; and transmitting the information to an output device thatprivately outputs information to the user.
 14. The method of claim 13,wherein generating the search query based on the speech occurring priorto determining that the user is thinking the key thought comprises:converting the speech to text; and identifying one or more questions inthe text.
 15. The method of claim 13, further comprising: detecting asecond neural activity of the user; determining that the user isthinking a second key thought based on the detected second neuralactivity; and in response to determining that the user is thinking asecond key thought, performing one of a fact-checking functions onspeech occurring prior to determining that the user is thinking thesecond key thought or recording speech occurring prior to determiningthat the sure is thinking the second key thought.
 16. A system,comprising: a neural sensor included in a head-worn assembly; and aprocessor coupled to the neural sensor and configured to: detect speech;detect a first neural activity of a user via the neural sensor thatmeasures neural activity of one or more brain regions of the user;determine, via a neural analysis application, that the user is thinkinga key thought based on the detected first neural activity, wherein theneural analysis application analyzes the first neural activity todetermine that the first neural activity is associated with the userthinking the key thought, wherein the neural analysis application isconfigured to differentiate the first neural activity from other neuralactivity of the user; in response to determining that the user isthinking the key thought, generate a search query based on the speechoccurring during a predetermined time period prior to determining thatthe user is thinking the key thought; receive information based on thesearch query; and transmit the information to an output device.
 17. Thesystem of claim 16, wherein the neural sensor includes anelectroencephalography (EEG) device.
 18. The system of claim 16, whereinthe head-worn assembly comprises one of a headphone-based assembly or anearbud-based assembly.
 19. The system of claim 16, wherein the outputdevice comprises one of a loudspeaker included in the head-worn assemblyor a display device included in the head-worn assembly.